{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Troubleshooting SSL Issues with EdgarTools\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dgunning/edgartools/blob/main/notebooks/02_troubleshooting_ssl.ipynb)\n",
    "\n",
    "SSL (Secure Sockets Layer) certificate errors are common when working with SEC Edgar data, especially in corporate environments with VPNs or SSL inspection proxies. This notebook will help you diagnose and resolve these issues.\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "1. Using the built-in `configure_http()` function (⭐ Primary Solution)\n",
    "2. Running SSL diagnostics with `diagnose_ssl()`\n",
    "3. Understanding common error messages\n",
    "4. Handling corporate proxies and VPNs\n",
    "5. Jupyter notebook-specific challenges and solutions\n",
    "\n",
    "## Why Do SSL Errors Occur?\n",
    "\n",
    "SSL errors typically happen when:\n",
    "- **Corporate VPNs or proxies** intercept HTTPS traffic (SSL inspection)\n",
    "- Your Python environment can't verify the SEC's SSL certificate\n",
    "- System certificate stores are outdated or misconfigured\n",
    "- Certain cloud environments (Google Colab) have certificate issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Installation\n",
    "\n",
    "Make sure you have the latest version of EdgarTools (v4.34.0+) which includes SSL troubleshooting tools:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade edgartools -q\n",
    "print(\"EdgarTools installed/upgraded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Quick Test - Do You Have SSL Issues?\n",
    "\n",
    "Let's quickly check if you have SSL problems:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from edgar import set_identity, Company\n",
    "\n",
    "# Set your identity (required by SEC)\n",
    "set_identity(\"Your Name your.email@example.com\")\n",
    "\n",
    "# Try to fetch data\n",
    "try:\n",
    "    company = Company(\"AAPL\")\n",
    "    print(\"✅ SUCCESS! No SSL issues detected.\")\n",
    "    print(f\"Connected to: {company.name}\")\n",
    "    print(\"\\nYou can skip this notebook and continue using EdgarTools normally!\")\n",
    "except Exception as e:\n",
    "    print(\"❌ SSL Issue Detected!\")\n",
    "    print(f\"Error: {type(e).__name__}\")\n",
    "    print(f\"\\nContinue with this notebook to fix it.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. ⭐ Primary Solution: Using `configure_http()`\n",
    "\n",
    "**EdgarTools v4.34.0+** includes a `configure_http()` function specifically designed to handle SSL issues in corporate environments.\n",
    "\n",
    "### ⚠️ CRITICAL: Import Order Matters!\n",
    "\n",
    "You **MUST** call `configure_http()` **BEFORE** importing other edgar classes. This is especially important in Jupyter notebooks where cells might run out of order.\n",
    "\n",
    "### The Correct Pattern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✅ CORRECT - Configure FIRST\n",
    "from edgar import configure_http\n",
    "configure_http(verify_ssl=False)\n",
    "\n",
    "print(\"✓ HTTP client configured with SSL verification disabled\")\n",
    "print(\"Now you can import other edgar classes...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now import and use edgar classes\n",
    "from edgar import Company, set_identity\n",
    "\n",
    "set_identity(\"Your Name your.email@example.com\")\n",
    "\n",
    "try:\n",
    "    company = Company(\"MSFT\")\n",
    "    print(f\"✅ SUCCESS! Connected to {company.name}\")\n",
    "    print(f\"CIK: {company.cik}\")\n",
    "    \n",
    "    filings = company.get_filings(form=\"10-K\")\n",
    "    print(f\"✅ Retrieved {len(filings)} 10-K filings\")\n",
    "except Exception as e:\n",
    "    print(f\"❌ Still having issues: {e}\")\n",
    "    print(\"\\nTry running the diagnostic below...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ❌ Common Mistake: Wrong Import Order\n",
    "\n",
    "This will **NOT** work if you've already imported edgar classes:\n",
    "\n",
    "```python\n",
    "# ❌ WRONG - imports already happened\n",
    "from edgar import Company, configure_http\n",
    "configure_http(verify_ssl=False)  # Too late! Client already created\n",
    "```\n",
    "\n",
    "### 🔧 If You Made This Mistake in Jupyter:\n",
    "\n",
    "1. **Kernel → Restart Kernel** (clears all state)\n",
    "2. Run cells in the correct order from the top\n",
    "3. `configure_http()` FIRST, then other imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Running SSL Diagnostics\n",
    "\n",
    "EdgarTools includes a comprehensive diagnostic tool that checks your network configuration and provides specific recommendations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the built-in diagnostic\n",
    "from edgar import diagnose_ssl\n",
    "\n",
    "# This will display a comprehensive report\n",
    "result = diagnose_ssl(display=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Diagnostic Results\n",
    "\n",
    "The diagnostic checks:\n",
    "\n",
    "1. **Environment** - Python version, installed packages, certificate locations\n",
    "2. **Proxy Configuration** - Detects proxy environment variables\n",
    "3. **Network Tests**:\n",
    "   - DNS resolution for www.sec.gov\n",
    "   - TCP connection to port 443\n",
    "   - SSL handshake and certificate verification\n",
    "   - **Corporate proxy detection** - Identifies SSL inspection\n",
    "4. **HTTP Client State** - Checks if your configuration is applied correctly\n",
    "\n",
    "### Key Indicators:\n",
    "\n",
    "- ✅ **HTTP Request (Configured) - PASS** → `configure_http()` works!\n",
    "- ❌ **HTTP Request (Default) - FAIL** → Expected if you have SSL issues\n",
    "- ⚠️ **Client Settings Mismatch** → `configure_http()` called too late\n",
    "- 🔍 **Corporate Proxy Detected** → Your network intercepts HTTPS traffic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Corporate Proxy Configuration\n",
    "\n",
    "If the diagnostic detects a proxy or your IT department provides a proxy URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For corporate proxies\n",
    "from edgar import configure_http\n",
    "\n",
    "# Without authentication\n",
    "configure_http(\n",
    "    verify_ssl=False,\n",
    "    proxy=\"http://proxy.company.com:8080\"\n",
    ")\n",
    "\n",
    "# With authentication (uncomment if needed)\n",
    "# configure_http(\n",
    "#     verify_ssl=False,\n",
    "#     proxy=\"http://username:password@proxy.company.com:8080\"\n",
    "# )\n",
    "\n",
    "print(\"Proxy configured\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Platform-Specific Solutions\n",
    "\n",
    "### Google Colab\n",
    "\n",
    "Google Colab sometimes has certificate issues. The `configure_http()` approach works well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Detect if we're in Google Colab\n",
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "    print(\"✅ Running in Google Colab\")\n",
    "    \n",
    "    # Apply Colab-specific configuration\n",
    "    from edgar import configure_http\n",
    "    configure_http(verify_ssl=False)\n",
    "    \n",
    "    print(\"SSL verification disabled for Colab (safe in this environment)\")\n",
    "except ImportError:\n",
    "    IN_COLAB = False\n",
    "    print(\"Not running in Google Colab\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local Jupyter\n",
    "\n",
    "For local Jupyter installations, try updating certificates first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update certifi package\n",
    "!pip install --upgrade certifi -q\n",
    "\n",
    "import certifi\n",
    "print(f\"Certifi version: {certifi.__version__}\")\n",
    "print(f\"Certificate bundle: {certifi.where()}\")\n",
    "print(\"\\n⚠️ If issues persist, restart the kernel and use configure_http()\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Standalone Scripts (When Jupyter Gets Confusing)\n",
    "\n",
    "Jupyter notebooks can have hidden state that causes confusion with import order. EdgarTools provides standalone scripts that run in a fresh Python process:\n",
    "\n",
    "### Download and run these scripts:\n",
    "\n",
    "```bash\n",
    "# 1. Comprehensive diagnostic\n",
    "curl -O https://raw.githubusercontent.com/dgunning/edgartools/main/scripts/diagnose_ssl.py\n",
    "python diagnose_ssl.py\n",
    "\n",
    "# 2. Test SSL configuration\n",
    "curl -O https://raw.githubusercontent.com/dgunning/edgartools/main/scripts/test_edgar_ssl.py\n",
    "python test_edgar_ssl.py\n",
    "```\n",
    "\n",
    "### Why use standalone scripts?\n",
    "\n",
    "- **Fresh process** - No hidden Jupyter state\n",
    "- **Clear execution order** - No cell confusion\n",
    "- **Reproducible** - Same results every time\n",
    "- **Easy to share** - Send output to support\n",
    "\n",
    "See the complete guide: [scripts/README_SSL_TROUBLESHOOTING.md](https://github.com/dgunning/edgartools/blob/main/scripts/README_SSL_TROUBLESHOOTING.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Common Error Messages Explained\n",
    "\n",
    "### `SSLCertVerificationError: certificate verify failed`\n",
    "**Cause:** Corporate VPN/proxy is replacing SEC.gov's certificate (SSL inspection)\n",
    "\n",
    "**Solution:** `configure_http(verify_ssl=False)`\n",
    "\n",
    "### `Client settings mismatch detected`\n",
    "**Cause:** `configure_http()` called after HTTP client was already created\n",
    "\n",
    "**Solution:** \n",
    "1. Restart Jupyter kernel\n",
    "2. Call `configure_http()` FIRST\n",
    "3. Then import other edgar classes\n",
    "\n",
    "### `DNS resolution failed`\n",
    "**Cause:** Network connectivity issue or DNS configuration problem\n",
    "\n",
    "**Solution:** \n",
    "- Check internet connection\n",
    "- Verify www.sec.gov is accessible\n",
    "- Contact IT if on corporate network\n",
    "\n",
    "### `TCP connection failed`\n",
    "**Cause:** Firewall blocking port 443 to SEC.gov\n",
    "\n",
    "**Solution:** Contact IT department"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Comprehensive Connection Test\n",
    "\n",
    "Let's verify everything is working with a comprehensive test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from edgar import Company\n",
    "\n",
    "def test_connection():\n",
    "    \"\"\"Test connection to SEC Edgar with multiple operations\"\"\"\n",
    "    tests = [\n",
    "        (\"Company data\", lambda: Company(\"AAPL\")),\n",
    "        (\"Filings retrieval\", lambda: Company(\"AAPL\").get_filings(form=\"10-K\")),\n",
    "        (\"Filing details\", lambda: Company(\"AAPL\").get_filings(form=\"10-K\")[0]),\n",
    "    ]\n",
    "    \n",
    "    passed = 0\n",
    "    print(\"Running connection tests...\\n\")\n",
    "    \n",
    "    for name, test_fn in tests:\n",
    "        try:\n",
    "            result = test_fn()\n",
    "            print(f\"✅ {name}: PASSED\")\n",
    "            passed += 1\n",
    "        except Exception as e:\n",
    "            print(f\"❌ {name}: FAILED ({type(e).__name__})\")\n",
    "    \n",
    "    print(\"\\n\" + \"=\"*50)\n",
    "    print(f\"Results: {passed}/{len(tests)} tests passed\")\n",
    "    \n",
    "    if passed == len(tests):\n",
    "        print(\"\\n🎉 All tests passed! Your connection is working perfectly.\")\n",
    "    elif passed > 0:\n",
    "        print(\"\\n⚠️ Some tests failed. Review the diagnostic above.\")\n",
    "    else:\n",
    "        print(\"\\n❌ All tests failed. Run diagnose_ssl() for details.\")\n",
    "    \n",
    "    return passed\n",
    "\n",
    "test_connection()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Best Practices\n",
    "\n",
    "### ✅ DO\n",
    "\n",
    "1. **Call `configure_http()` first** - Before any other edgar imports\n",
    "2. **Restart Jupyter kernel** when troubleshooting\n",
    "3. **Run diagnostic** to understand your specific situation\n",
    "4. **Use standalone scripts** when Jupyter state is confusing\n",
    "5. **Set identity** with `set_identity()` before making requests\n",
    "\n",
    "### ❌ DON'T\n",
    "\n",
    "1. **Don't disable SSL verification in production** - Only for development/Colab\n",
    "2. **Don't store proxy credentials** in notebooks or code repositories\n",
    "3. **Don't run cells out of order** in Jupyter\n",
    "4. **Don't call `configure_http()` multiple times** - Once is enough\n",
    "\n",
    "### Environment-Specific Recommendations\n",
    "\n",
    "| Environment | Solution |\n",
    "|------------|----------|\n",
    "| Google Colab | `configure_http(verify_ssl=False)` |\n",
    "| Corporate VPN | `configure_http(verify_ssl=False)` |\n",
    "| Corporate Proxy | `configure_http(verify_ssl=False, proxy=...)` |\n",
    "| Local Jupyter | Update certifi, or use `configure_http()` |\n",
    "| Production | **Never** disable SSL verification |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Quick Reference: Python Scripts\n",
    "\n",
    "### For Your Own Scripts\n",
    "\n",
    "```python\n",
    "# my_edgar_script.py\n",
    "from edgar import configure_http\n",
    "configure_http(verify_ssl=False)\n",
    "\n",
    "from edgar import Company, set_identity\n",
    "set_identity(\"Your Name your@email.com\")\n",
    "\n",
    "company = Company(\"AAPL\")\n",
    "print(company.name)\n",
    "```\n",
    "\n",
    "### For Jupyter Notebooks\n",
    "\n",
    "```python\n",
    "# Cell 1 - Run FIRST\n",
    "from edgar import configure_http\n",
    "configure_http(verify_ssl=False)\n",
    "\n",
    "# Cell 2 - Use edgar normally\n",
    "from edgar import Company, set_identity\n",
    "set_identity(\"Your Name your@email.com\")\n",
    "\n",
    "company = Company(\"AAPL\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Getting More Help\n",
    "\n",
    "If you're still experiencing issues:\n",
    "\n",
    "### 1. Run the Full Diagnostic\n",
    "\n",
    "```python\n",
    "from edgar import diagnose_ssl\n",
    "result = diagnose_ssl(display=False)  # Get structured data\n",
    "\n",
    "# Check specific issues\n",
    "if result.network_tests.ssl_handshake.is_corporate_proxy:\n",
    "    print(\"Corporate SSL inspection detected\")\n",
    "    \n",
    "if not result.http_client_state.settings_match:\n",
    "    print(\"Call configure_http() earlier in your code\")\n",
    "```\n",
    "\n",
    "### 2. Use Standalone Scripts\n",
    "\n",
    "Download and run:\n",
    "- [diagnose_ssl.py](https://github.com/dgunning/edgartools/blob/main/scripts/diagnose_ssl.py)\n",
    "- [test_edgar_ssl.py](https://github.com/dgunning/edgartools/blob/main/scripts/test_edgar_ssl.py)\n",
    "\n",
    "### 3. Report Issues\n",
    "\n",
    "When reporting issues, include:\n",
    "- Output from `diagnose_ssl()`\n",
    "- Python version (`python --version`)\n",
    "- EdgarTools version (`pip show edgartools`)\n",
    "- Operating system\n",
    "- Whether you're behind a VPN/proxy\n",
    "\n",
    "**GitHub Issues:** [https://github.com/dgunning/edgartools/issues](https://github.com/dgunning/edgartools/issues)\n",
    "\n",
    "### 4. Additional Documentation\n",
    "\n",
    "- [SSL Troubleshooting Guide](https://github.com/dgunning/edgartools/blob/main/scripts/README_SSL_TROUBLESHOOTING.md)\n",
    "- [EdgarTools Documentation](https://github.com/dgunning/edgartools)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "### The Solution (90% of cases):\n",
    "\n",
    "```python\n",
    "# Step 1: Configure FIRST\n",
    "from edgar import configure_http\n",
    "configure_http(verify_ssl=False)\n",
    "\n",
    "# Step 2: Use edgar normally\n",
    "from edgar import Company, set_identity\n",
    "set_identity(\"Your Name your@email.com\")\n",
    "company = Company(\"AAPL\")\n",
    "```\n",
    "\n",
    "### Remember:\n",
    "\n",
    "1. **Import order matters** - `configure_http()` must be FIRST\n",
    "2. **Restart Jupyter kernel** when troubleshooting\n",
    "3. **Use diagnostic tools** - `diagnose_ssl()` tells you exactly what's wrong\n",
    "4. **SSL verification is important** - Only disable in safe environments\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "Now that your SSL issues are resolved:\n",
    "- **Notebook 01**: Getting Started with EdgarTools\n",
    "- **Notebook 03**: Working with Financial Data\n",
    "- **Notebook 04**: Company Search and Filings"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
